Review: coder_eval — pr:141 (2 files) axis:1,2,3,4,5,6,7,8

Scope: pr:141 (2 files) axis:1,2,3,4,5,6,7,8 · branch chore/plugin-discovery-metadata · c0156b4 · 2026-08-27T19:53Z · workflow variant

Change class: trivial — plugin/marketplace manifest metadata only (descriptions, keywords, $schema, discovery fields); no Python, control flow, or persisted-schema semantics touched

The harness core is in excellent shape — perfect scores on type safety, tests, security, architecture, error handling, API surface, and evaluation-harness quality, with zero critical/high/medium findings and nothing that can change a task's score or final_status for identical agent output — so the only real risk is editorial drift in the newly duplicated plugin discovery metadata (keywords vs tags split across two manifests with no rule and no parity test), which costs nothing today but rots quietly; bottom line: ship it, and spend the small remaining effort on making the two manifests a single enforced surface.

Summary

Overall Score: 10 / 10 · Weakest Axis: Code Quality & Style at 9.8 / 10
Totals: 🔴 0 · 🟠 0 · 🟡 0 · 🔵 2 across 8 axes.

Blockers

None.

Non-blocking, but please consider before merge

None.

Nits

1. [Axis 1] tags added alongside keywords on the marketplace entry: 12 discovery strings split across two synonymous fields with no rule, disjoint values, and no counterpart in plugin.json (.claude-plugin/marketplace.json:29) — Line 29 adds "tags": ["skills", "evals", "github-actions", "sandbox", "llm-judge"] directly under the 7-item "keywords" block (lines 20-28). The pinned schema (fetched from https://www.schemastore.org/claude-code-marketplace.json) describes the two fields near-identically — keywords: 'Tags for plugin discovery and categorization'; tags: 'Tags for searchability and discovery' — so there is no consumer distinction to key the split on, and the two value sets are fully disjoint (set(tags) & set(keywords) == set()) while containing synonyms across the boundary: evals/evaluation, and skills against three keyword variants of the same concept (claude-code-skills, skill-testing, skill-activation). The asymmetry is the concrete cost: the plugin-manifest schema has no tags property at all (top-level props are $schema, agents, author, channels, commands, dependencies, description, homepage, hooks, keywords, license, lspServers, mcpServers, monitors, name, outputStyles, repository, settings, skills, themes, userConfig, version), so plugins/coder-eval/.claude-plugin/plugin.json cannot carry the five tags values — an installed user's manifest loses them, and a future editor has no rule for whether a new term belongs in keywords or tags. Recommendation: drop tags and fold any genuinely distinct terms into the single keywords list that both manifests share (which also removes five more strings from the parity surface in finding #1), or add a one-line comment/doc note stating the split rule if both are kept deliberately.
2. [Axis 1] Newly pinned $schema does not define displayName, the other key added in the same commit, and nothing validates either file against the pin (plugins/coder-eval/.claude-plugin/plugin.json:4) — The same commit adds "$schema": "https://json.schemastore.org/claude-code-plugin-manifest.json" (line 2) and "displayName": "Coder Eval", (line 4) — but I fetched that schema (301 → https://www.schemastore.org/claude-code-plugin-manifest.json, $comment: Generated on 2026-04-23) and displayName is not among its properties; same for the marketplace item schema, where .claude-plugin/marketplace.json:12 adds displayName to the plugin entry. Because neither schema sets additionalProperties, both files validate with 0 errors (confirmed with jsonschema.Draft7Validator), so the field is tolerated but uncovered: an editor gives no completion or hover doc for it, and a typo like displyName would also pass the pinned schema silently. The pin is also purely editorial — .github/workflows/pr-checks.yml:241,244 gates on claude plugin validate --strict, which is stricter than the schema (I verified locally that it emits Unknown field 'totallyBogusKey'. Claude Code ignores it at load time. and exits 1, while accepting displayName), and no workflow or Makefile target references jsonschema/check-jsonschema/ajv/schemastore. Recommendation: keep the pin for editor UX but note in the PR that enforcement stays with claude plugin validate --strict, and either raise a schemastore PR adding displayName or drop the field until the pinned schema covers it — otherwise the pin implies validation coverage it does not give for the very key added beside it.

What's Missing

Parallel paths:

• 🟡 🟡 Both manifests were re-led with "Test whether your Claude Code skills actually trigger — then author, run, and analyze…", but every prose surface a user actually lands on still leads with the inverted old order: plugins/coder-eval/README.md:3 ("Author, run, and analyze Coder Eval suites from inside Claude Code — including whether your own Claude Code skills actually trigger"), docs/PLUGIN.md:2-5 (frontmatter description), and README.md:100-113 (§ Use inside Claude Code). The plugin README is the page the marketplace one-liner leads to, so the new positioning survives exactly one click. pyproject.toml:4 (the PyPI one-liner) is a fourth copy, also unchanged — decide whether it is deliberately audience-specific or also stale. (trigger: plugins/coder-eval/.claude-plugin/plugin.json)

Tests:

• 🟡 🟡 The commit makes the two manifests carry six byte-identical fields (description, keywords, author, homepage, repository, license) and nothing keeps them in parity: grep over tests/, Makefile and .github/workflows/ finds .claude-plugin referenced only by tests/test_action_version_pin.py (version) and tests/test_verify_published_workflow.py (a release-step name), and no test or CE rule mentions keywords/description for either file. The repo already institutionalizes exactly this shape for version (a derived pin with an anchor test); the new duplicated block ships with none. A CE-style test asserting marketplace.plugins[0][k] == plugin.json[k] for the shared keys is ~10 lines and would make the next one-sided edit fail in make lint. (trigger: .claude-plugin/marketplace.json)
• 🔵 🔵 No test or CI step validates either JSON against the $schema it now declares — no jsonschema/check-jsonschema/ajv reference exists in the repo, so the pin is editor-only. A pytest that loads both files and validates them against a vendored copy of the two schemas (or an offline JSON-parse + required-key assertion) would make the pin load-bearing and would not depend on network or on the Claude CLI. (trigger: plugins/coder-eval/.claude-plugin/plugin.json) (restates: Axis 1: Newly pinned $schema does not define displayName)

Downstream consumers:

• 🔵 🔵 The sole gate on the newly added keys is claude plugin validate --strict in .github/workflows/pr-checks.yml:240-244, whose validator is installed unpinned (npm install -g @anthropic-ai/claude-code, line 233). displayName (both files) and tags (marketplace) are accepted by today's CLI; whichever CLI ships on a future run decides whether they still are, so a metadata-only change can turn an unrelated PR red with no repo-side signal. Pin the CLI version (or add the offline schema test above) so the acceptance criterion for these keys is versioned. (trigger: .claude-plugin/marketplace.json)

Display & mapping dicts:

• 🔵 🔵 The generated docs-index blurb for the plugin page still reads "Install the Claude Code plugin — author, run, and analyze suites from inside the agent" — mkdocs.yml:92 (the SSOT) mirrored into README.md:218, docs/index.md:90 and docs/llms.txt:37. CE028 only enforces that the three generated tables match mkdocs.yml, so it cannot see that the blurb now contradicts the manifests; updating means editing mkdocs.yml then running make docs-indexes (never hand-editing the tables). (trigger: plugins/coder-eval/.claude-plugin/plugin.json)

Daily/nightly:

• 🔵 🔵 Rollout skew is unstated. marketplace.json is served unversioned from main, so /plugin marketplace add browsers see the new description immediately; plugin.json's version is a derived pin of pyproject (0.11.3, correctly untouched here and guarded by tests/test_action_version_pin.py), and per that test's own docstring Claude Code keys plugin updates off it — so already-installed users keep the old description/keywords until the next release. The PR should say the plugin-side half of this metadata goes live only at the next release, which is also the window in which the two duplicated descriptions are observably different in the wild. (trigger: plugins/coder-eval/.claude-plugin/plugin.json)
• 🔵 🔵 No post-release or nightly job covers the plugin manifests. plugin-validate lives in pr-checks.yml (pull_request/push/merge_group only), while .github/workflows/verify-published-action.yml — the nightly + post-Release parity job — verifies the GitHub Actions Marketplace listing, tag, pin and PyPI, and never touches /plugin marketplace add UiPath/coder_eval. This change expands the plugin's public discovery surface without extending that verification, so a manifest that stops installing after a release is only noticed by users. (trigger: .claude-plugin/marketplace.json)

Harness & Lint Improvements

Static checks (lint / type):

• [ce-lint] New rule CE044 — plugin/marketplace manifest metadata parity (next free number; highest in tree is CE043). Statement: the marketplace plugin entry and the plugin manifest it points at must carry identical values for every field both manifests can express, and the marketplace entry must not carry a discovery field the plugin manifest cannot mirror. Implementation: a whole-tree JSON rule (not a per-file .py AST BaseRule), so wire it like CE026/CE028/CE033 as a dedicated @pytest.mark.lint test class in tests/test_custom_lint.py backed by a helper module tests/lint/plugin_manifest_parity.py. It (a) parses /Users/religa/src/coder_eval/.claude-plugin/marketplace.json, resolves each entry's source (./plugins/coder-eval) to a real directory containing .claude-plugin/plugin.json (fails if it does not resolve), (b) asserts byte-equality of the shared keys name, description, author, homepage, repository, license, keywords, and (c) fails on any extra discovery-string key on the marketplace entry (tags, and anything future) unless it is listed in an explicit MARKETPLACE_ONLY = {"key": "reason"} allowlist in the helper — the allowlist entry is the missing "rule for where a new term goes", recorded in code instead of tribal knowledge. Prevents: Finding #1 directly: "tags": ["skills", "evals", "github-actions", "sandbox", "llm-judge"] at .claude-plugin/marketplace.json:29 has no counterpart property in the plugin manifest, so those five discovery strings are dropped from an installed user's copy, and the split against the 7-item keywords block (lines 20-28) has no stated rule while holding cross-boundary synonyms (evals/evaluation, skills/claude-code-skills). It also prevents the untested drift class the same shape invites: editing description/keywords/homepage in one manifest only — today nothing in tests/ compares the two files (only tests/test_action_version_pin.py touches plugin.json, and only its version).
• [ce-lint] New rule CE045 — a pinned $schema must actually be enforced, closed-world. Statement: every JSON file in the repo that declares a $schema must validate against a vendored copy of that exact schema, and every key it uses (top-level, plus each plugins[] entry for the marketplace file) must be present in that schema's properties unless allowlisted with a reason. Implementation: vendor the two documents as tests/lint/schemas/claude-code-plugin-manifest.json and tests/lint/schemas/claude-code-marketplace.json, then add a @pytest.mark.lint class over a helper tests/lint/schema_pin_coverage.py that runs jsonschema (already a runtime dep, pyproject.toml:50 — no new dependency, no network) and then the extra closed-world key check, with UNCOVERED_KEYS = {("plugin.json", "displayName"): "<reason>"}. The closed-world half is the load-bearing part: neither schemastore document sets additionalProperties, so plain validation returns 0 errors for any key. Prevents: Finding #2: plugins/coder-eval/.claude-plugin/plugin.json:2 pins https://json.schemastore.org/claude-code-plugin-manifest.json while line 4 adds "displayName": "Coder Eval", which that schema does not define (same for displayName on the marketplace entry at .claude-plugin/marketplace.json:12) — and nothing in Makefile, tests/, or .github/workflows/ validates either file against the pin, so both the uncovered key and a typo like displyName pass silently. Under CE045 the uncovered key becomes a lint failure that must be either dropped or allowlisted with a written reason, and a misspelling of any covered key fails outright without waiting for the CI-only claude plugin validate --strict step.

Harness improvements (not statically reachable):

• Vendor the two schemastore documents under tests/lint/schemas/ (the prerequisite CE045 needs to run offline) and add a make plugin-schemas-refresh target plus a scheduled workflow job that re-fetches both URLs and fails if the vendored copy is stale. Note in the target that json.schemastore.org 301-redirects to www.schemastore.org and that the plugin-manifest document is dated ($comment: Generated on 2026-04-23), so "the pin is current" is a fact with an expiry date. Why not static: Freshness of an externally versioned document can only be established by fetching it. The make lint gate must stay hermetic and offline (it runs in every make verify), so the network half has to be a scheduled job while the lint half reads the vendored snapshot. Prevents: Finding #2's underlying gap — the pin implies coverage it does not give. When upstream adds displayName (or the team raises the schemastore PR), the refresh job is what tells us the allowlist entry can be deleted, instead of the exemption calcifying.
• Pin the validator in the plugin-validate job: .github/workflows/pr-checks.yml:238 runs an unpinned npm install -g @anthropic-ai/claude-code (as do 6 other jobs), so the strictness of the only real manifest gate floats with whatever the registry serves that morning. Pin an exact version (renovate/dependabot-bumped) and record in the job comment that claude plugin validate --strict — not the $schema pin — is the enforcement authority for these two files. Why not static: The gate's strength is a property of an external binary's runtime behavior; no AST or schema check can tell whether the installed CLI still rejects an unknown key. Prevents: Finding #2's expectation mismatch (the newly added $schema pin reads as validation coverage while the actual gate lives in CI), and the silent weakening of that gate across CLI releases.
• Add a canary step to the same plugin-validate job: copy plugins/coder-eval/.claude-plugin/plugin.json to a temp dir, inject a bogus key, and assert claude plugin validate --strict exits non-zero. Today the job only proves the good manifests pass, which is also what a validator that silently stopped enforcing --strict would report. Why not static: Requires executing the external CLI against a mutated fixture; the assertion is about observed exit status, not about repository text. Prevents: Silent regression of the unknown-field / typo class that finding #2 rests on (displyName passing unnoticed), and by extension the tags-style unmirrored-field class from finding #1 if a future CLI version starts flagging it.

Top 5 Priority Actions

1. No finding affects scoring or final_status, so start with the only structural gap: collapse the synonymous tags list at .claude-plugin/marketplace.json:29 into the single keywords block above it (lines 20-28), since the values are fully disjoint yet semantically overlapping (evals/evaluation, skills vs three keyword variants) and tags has no counterpart property in the plugin-manifest schema, so an installed user's copy silently loses those five terms.
2. Add a parity test (mirroring tests/test_action_version_pin.py) asserting that the shared discovery fields — description, keywords, author, homepage, repository, license — are byte-identical between .claude-plugin/marketplace.json:14-29 and plugins/coder-eval/.claude-plugin/plugin.json:6-19, because nothing today catches a one-sided edit to either file.
3. Decide the fate of displayName at plugins/coder-eval/.claude-plugin/plugin.json:4 (and .claude-plugin/marketplace.json:12): it is absent from both pinned schemas, so an editor gives no completion and a typo like displyName validates clean — either raise a schemastore PR adding it or drop the key until the pin covers it.
4. Record in the PR description (or a comment near plugins/coder-eval/.claude-plugin/plugin.json:2) that the new $schema pins are editorial-only and that real enforcement stays with claude plugin validate --strict at .github/workflows/pr-checks.yml:241,244, so a future reader does not assume the pin gates CI.
5. If both keywords and tags are kept deliberately, write the one-line split rule into CLAUDE.md's plugin section so the next editor has a deterministic place to put a new discovery term — otherwise the two lists diverge on the next skill added.

Stats: 0 🔴 · 0 🟠 · 0 🟡 · 2 🔵 across 8 axes reviewed.

Review: coder_eval — pr:141

Scope: pr:141 · branch chore/plugin-discovery-metadata · c0156b4 · 2026-08-27T20:09Z · workflow variant

Change class: trivial — metadata/manifest-only edit: two JSON discovery manifests (description text, keywords/tags, displayName, $schema pins); no Python, no code path, no schema consumed by the harness

The change is safe and well scoped — nothing here touches the harness, so no task's score or final_status can move for identical agent output (Security, Architecture, Error Handling and Evaluation Harness are all clean at 10/10) — and the only real risk is maintenance drift in the plugin's public shopfront, where eight metadata fields are now hand-synced byte-for-byte across two manifests with no parity guard (the pair already drifted on keywords on main), the repositioned lede was not rippled to the README and docs surfaces that mirror it, and two new $schema pins declare a contract no test checks; bottom line: ship it, then add the ~15-line parity test the repo's own test_action_version_pin.py precedent calls for.

Anchor note. This review was scoped at c0156b4. While it ran, the PR
advanced two commits to 78de45e (chore(plugin): give the marketplace owner a contact address, chore(plugin): give both author objects the same contact address). Those commits only add "email": "coder-eval@uipath.com"
to the marketplace owner and to both author objects — no finding below is
invalidated, and finding 1 is strengthened: author is now a 9th
byte-identical hand-synced field across the two manifests. All file:line
citations below have been re-anchored to 78de45e.

Coverage caveat. Three sub-agents hit a session limit. Axis 2 (Type
Safety) produced no results and is excluded from the report and from the
Overall mean — read its absence as not reviewed, not as clean. The verify
agents for Axes 5 and 7 also failed; their surviving medium+ finding was
merged into finding 1, whose verifier did run. Overall is the mean over the
7 axes that reported.

Summary

Overall Score: 9.9 / 10 · Weakest Axis: Code Quality & Style at 9.5 / 10
Totals: 🔴 0 · 🟠 0 · 🟡 1 · 🔵 4 across 7 axes.

Blockers

None.

Non-blocking, but please consider before merge

1. [Axis 1] Manifest metadata duplicated verbatim across marketplace.json and plugin.json with no parity guard (widened from 3 to 8 shared keys; the pair already drifted on keywords) (.claude-plugin/marketplace.json:15) — The PR grows the hand-synced surface between .claude-plugin/marketplace.json's plugin entry and plugins/coder-eval/.claude-plugin/plugin.json from 3 fields to 8. Verified byte-identical after this change: displayName (marketplace.json:13 / plugin.json:4), description (marketplace.json:15 / plugin.json:6 — both exactly "Test whether your Claude Code skills actually trigger — then author, run, and analyze the eval suite that proves it, locally or as a CI gate.", 141 chars), author (:16 / :7), homepage (:17 / :8), repository (:18 / :9), license (:19 / :10), keywords (:21-29 / :11-19), plus name.

This is not a hypothetical drift risk — it already happened on main: git show origin/main:.claude-plugin/marketplace.json has "keywords": ["evaluation", "testing", "skills", "benchmark", "ci"] while git show origin/main:plugins/coder-eval/.claude-plugin/plugin.json has ["evaluation", "testing", "claude-code-skills", "benchmark", "ci"]. This PR silently repairs that drift without adding anything that stops it recurring across 8 fields instead of 3.

The drift is user-visible and asymmetric: strict defaults to true in the marketplace-entry schema ("Require the plugin manifest to be present in the plugin folder"), so the marketplace entry's copy is what a user sees while BROWSING the catalog and plugin.json's copy is what they see once INSTALLED — a divergence shows the product's front door describing itself two different ways.

Nothing guards cross-manifest parity. (Corrected during verification: the plugin-validate job in .github/workflows/pr-checks.yml:218-243 DOES run claude plugin validate . --strict and claude plugin validate ./plugins/coder-eval --strict on every PR, so each file is schema-validated on its own and a JSON syntax error does NOT ship. What is unguarded is only the field-equality check between the two manifests — claude plugin validate never compares them.) This directly contradicts the repo's own stated convention — tests/test_action_version_pin.py exists verbatim because "Nothing asserted it, which is how action.yml shipped pinned to 0.8.6 while main was already 0.8.9" — and CLAUDE.md's harness-loop rule ("could a custom lint rule have prevented this?").

Fix: add a ~15-line test beside tests/test_action_version_pin.py that json.loads both files, locates the entry with name == "coder-eval" in plugins, and asserts equality for the shared key set (displayName, description, author, homepage, repository, license, keywords) — deliberately excluding version (marketplace entry has none), source, category, and tags (marketplace-entry-only per the runtime schema). That single test closes the parity gap (the only unguarded half).

Nits

1. [Axis 3] Newly pinned $schema URLs are unverified by any test or CI step, and already disagree with the runtime validator over the displayName key this PR adds (.claude-plugin/marketplace.json:2) — The PR adds "$schema": "https://json.schemastore.org/claude-code-marketplace.json" (marketplace.json:2) and "$schema": "https://json.schemastore.org/claude-code-plugin-manifest.json" (plugin.json:2), but grep -rn 'schemastore\|\$schema' tests/ .github/ Makefile returns ZERO hits — no test or CI step fetches either URL or validates the manifests against it, so the pin is editor-only and a typo'd or later-dead URL would ship unnoticed. That the two validators already disagree is demonstrable today: claude plugin validate --strict recognises displayName (a probe with "displaName" emits Unknown field 'displaName' — did you mean 'displayName'?), yet in both downloaded schemas displayName appears at exactly one path — /properties/channels/items/properties (plugin schema) and /properties/plugins/items/properties/channels/items/properties (marketplace schema) — i.e. neither declares displayName at the plugin-manifest top level or at plugins[]., so the two displayName keys this PR adds (marketplace.json:13, plugin.json:4) are outside the pinned schemas' documented property set and pass only because additionalProperties is unset in both. I confirmed both files currently validate clean (jsonschema.Draft7Validator → 0 errors each), so this is a missing guard, not a live breakage. Either add a network-optional test that Draft7-validates each manifest against a vendored copy of its declared $schema (skip on fetch failure, following the repo's derived-pin test style), or drop the $schema lines rather than declaring a contract nothing checks.
2. [Axis 7] Newly added keywords/tags discovery vocabulary adds no discovery value — inert for the client search, and internally redundant (near-synonyms, second vocabulary with no plugin.json counterpart) (.claude-plugin/marketplace.json:21) — The PR reworks keywords from 5 to 7 values and adds a new 5-value tags array (marketplace.json:21-30):

"keywords": ["claude-code-skills","skill-testing","skill-activation","evaluation","benchmark","agent-testing","ci"],
"tags": ["skills", "evals", "github-actions", "sandbox", "llm-judge"]

Neither array is consulted by the client's discovery search. The plugin browse/install filter in Claude Code 2.1.247 is, verbatim from the binary:

l.filter((W)=>W.entry.name.toLowerCase().includes(ae)||W.entry.displayName?.toLowerCase().includes(ae)||W.entry.description?.toLowerCase().includes(ae)||s&&W.marketplaceName.toLowerCase().includes(ae))

Only name, displayName, description and the marketplace name, matched as case-insensitive substrings. Corroborating de-facto evidence: of the 289 plugin entries in the installed claude-plugins-official marketplace, exactly 1 sets keywords, and all 3 that set tags use it as a curation label (["community-managed"]), not as topical keywords.

Two consequences worth recording. First, dropping testing and skills from keywords is NOT a discoverability regression — the field was never searched. Second, the levers this PR moved that DO work are description (mkt:15) and the newly added displayName (mkt:13 / plg:4), and both moved in the right direction: the new description adds the substrings test, trigger, eval suite and ci gate, which the base description did not contain.

Recommendation: keep both arrays (they cost nothing and may be indexed by an out-of-client directory), but do not treat them as the discovery lever, and add a short JSON-adjacent note or commit-message line saying so — otherwise the next contributor tunes keywords expecting search impact it cannot have. If a term must be findable in-client, it has to appear as a substring of description, displayName or name.
3. [Axis 7] Repositioned lede not rippled to the two prose surfaces that mirror the manifest description (plugins/coder-eval/.claude-plugin/plugin.json:6) — The stated intent of this PR (commit 8d7eba9, "lead both manifests with skill evaluation") is to invert the pitch so skill-activation testing comes first. plugin.json:6 and marketplace.json:14 now read "Test whether your Claude Code skills actually trigger — then author, run, and analyze the eval suite that proves it, locally or as a CI gate."

The two prose surfaces that mirrored the old manifest description were not updated and still lead with the old ordering:

• plugins/coder-eval/README.md:3-4: Author, run, and analyze [Coder Eval](https://coder-eval.com) suites from inside Claude / Code — including whether your own Claude Code skills actually trigger. — this is the base plugin.json description rendered as prose, verbatim in structure.
• docs/PLUGIN.md:2-5 frontmatter: description: >- / Install the Coder Eval plugin for Claude Code — six slash commands to scaffold, / author, review, run and analyze evaluation suites, including an activation suite / that measures whether your own Claude Code skills actually trigger.

A user who reads the marketplace listing (skill-testing first) then clicks through to the plugin README on GitHub or to coder-eval.com/PLUGIN gets the inverted pitch, with skill activation demoted to a trailing "including..." clause. Since plugins/coder-eval/README.md is one of the four SKILL_DOC_SURFACES (tests/test_custom_lint.py:1238) the repo already treats as a governed surface, the fix is cheap: rewrite the README:3-4 lede and the docs/PLUGIN.md:3-5 frontmatter to lead with skill-activation testing, matching the manifests.
4. [Axis 7] Listing description leads with a capability that needs an out-of-band CLI install and paid credentials, with no hint in the 139 chars available (plugins/coder-eval/.claude-plugin/plugin.json:6) — plugin.json:6 (and its twin marketplace.json:14) reads "Test whether your Claude Code skills actually trigger — then author, run, and analyze the eval suite that proves it, locally or as a CI gate." — an imperative capability claim as the lead. The plugin ships no runtime: four of its six skills shell out to a coder-eval binary that the plugin does not install, and running a suite additionally needs ANTHROPIC_API_KEY and real model spend.

The plugin's own skills treat this as the top user surprise and open by defending against it: plugins/coder-eval/skills/init/SKILL.md:17 — "Run coder-eval --version. Installing this plugin did not install the CLI, and"; plugins/coder-eval/skills/check-skill/SKILL.md:61 — same sentence; plugins/coder-eval/skills/task/SKILL.md:20 — "Installing this plugin did not install the CLI." docs/PLUGIN.md gives it a dedicated ## Prerequisite heading in bold.

This is not a truthfulness defect and it is not a regression (the base description had the same shape), which is why it is filed low. But the marketplace listing is frequently the only text a prospective user reads, and there is budget to say so: at 139 chars this description sits at the measured median (135) across the 147 plugin entries installed under ~/.claude/plugins/marketplaces/, where the max observed is 586. A trailing clause such as "…locally or as a CI gate. Drives the separately installed coder-eval CLI." costs ~45 chars, sets the expectation before install, and — per the search predicate documented in the keywords/tags finding — adds cli and install as searchable substrings for free.

What's Missing

Parallel paths:

• 🟡 Parallel paths 🟡 — .claude-plugin/marketplace.json's plugins[0] entry gained displayName/author/homepage/repository/license so it now hand-mirrors plugins/coder-eval/.claude-plugin/plugin.json on 8 keys (was 3), but no generator or parity test was added in the same change. claude plugin validate --strict (pr-checks.yml plugin-validate) validates each file against its own schema and never compares the two — I re-ran both locally and both pass, so the schema gate is genuinely blind to this. The 3-key surface had already drifted on keywords on origin/main; the PR repairs that drift silently and widens the surface that can drift next. (trigger: .claude-plugin/marketplace.json) (restates: Axis 1: Manifest metadata duplicated verbatim across marketplace.json and plugin.json with no parity guard)
• 🔵 Parallel paths 🔵 — the skills-first lede was not rippled, and the un-rippled set is larger than the two surfaces already filed: beyond plugins/coder-eval/README.md:3-4 and docs/PLUGIN.md:2-5, the root README.md:100-102 plugin pitch ("scaffold a suite, author a task, check whether a skill triggers, read the results" — skill-checking third) and docs/tutorials/07-plugin-in-claude-code.md:2-4 frontmatter ("install the plugin, scaffold a task directory, author and review a task, then read the run" — skill activation absent entirely) both still carry the old ordering. Four prose surfaces, not two. (trigger: plugins/coder-eval/.claude-plugin/plugin.json) (restates: Axis 7: Repositioned lede not rippled to the two prose surfaces that mirror the manifest description)
• 🔵 Parallel paths 🔵 — the marketplace-level description (.claude-plugin/marketplace.json:9, a distinct string from the plugin entry's) was repositioned to lead with skill testing, but its three peer product one-liners were not: pyproject.toml:4 ("Evaluate, benchmark, and A/B-test AI coding agents…" — the PyPI listing, the highest-traffic install surface), mkdocs.yml:2-5 site_description ("Evaluate & benchmark AI coding agents and Claude Code skills…" — the coder-eval.com meta description / search snippet) and README.md:1's H1 tagline. The GitHub repo description already leads skills-first ("Test that your Claude Code skills, MCP servers, and CLIs actually work…"), so after this PR the manifests + GitHub agree and those three do not. (trigger: .claude-plugin/marketplace.json)

Tests:

• 🔵 Tests 🔵 — no test ties marketplace.json's name + plugins[0].name to the install command five docs surfaces hardcode (README.md:107, plugins/coder-eval/README.md:14, docs/PLUGIN.md:18, docs/USER_GUIDE.md:134, docs/tutorials/07-plugin-in-claude-code.md:41 all say /plugin install coder-eval@coder-eval). This PR adds displayName: "Coder Eval" right beside name, which makes a follow-up "rename it to match the display name" edit the natural next move — and claude plugin validate --strict passes on a rename, so five copy-paste install commands would go stale with no gate firing. (trigger: .claude-plugin/marketplace.json)
• 🔵 Tests 🔵 — the two newly pinned $schema URLs (marketplace.json:2, plugin.json:2) declare a contract that nothing in tests/, Makefile or .github/ fetches or validates against; the manifests are schema-checked only by claude plugin validate --strict, which uses the CLI's own built-in rules, not the pinned schemastore documents. Either vendor the schemas and add a network-optional Draft7 validation test, or drop the pins. (trigger: plugins/coder-eval/.claude-plugin/plugin.json) (restates: Axis 3: Newly pinned $schema URLs are unverified by any test or CI step)

Display & mapping dicts:

• 🔵 Display / mapping 🔵 — the new displayName: "Coder Eval" (marketplace.json:13, plugin.json:4) becomes the title Claude Code renders in the plugin browser, but every in-repo title surface still renders coder-eval: the shields badge label claude__code__plugin-coder--eval (README.md:5), plugins/coder-eval/README.md:1 (# coder-eval — Claude Code plugin) and docs/PLUGIN.md's H1. CE026 already enforces exactly this class of parity for the GitHub Action (action.yml name: ↔ marketplace slug ↔ badge label); no analogous check exists for the plugin, so the client and the docs can now show two different product names. (trigger: .claude-plugin/marketplace.json)

Downstream consumers:

• 🔵 Downstream consumers 🔵 — the discovery vocabulary now diverges three ways with no shared source: the manifests (7 keywords + 5 tags, this PR), pyproject.toml:9-16 (27 PyPI keywords — has agent-testing/claude-code-skills but not the PR's new skill-testing/skill-activation), and the GitHub repo topics (20 — already carry skill-testing, so they are ahead of PyPI). If the repositioning is real it should move pyproject.toml's keywords and description in the same commit, since PyPI is the surface pip install coder-eval users actually search. Note also that the manifests' new vocabulary drops mcp and any cli term, which the repo's own GitHub description leads with ("skills, MCP servers, and CLIs"). (trigger: plugins/coder-eval/.claude-plugin/plugin.json) (restates: Axis 7: Newly added keywords/tags discovery vocabulary adds no discovery value)

Daily/nightly:

• 🔵 Daily / nightly 🔵 — blast radius on the two published-surface pipelines is unstated. claude plugin validate --strict runs only in pr-checks.yml's plugin-validate job (PR / push / merge_group); the nightly verify-published-action.yml (cron 17 6 * * *) checks tag/pin/PyPI/Marketplace parity for the Action only and never touches the plugin marketplace, so nothing post-merge confirms /plugin marketplace add UiPath/coder_eval still resolves or that the two schemastore URLs still exist. Two things I verified so the PR body can say them outright: both manifests pass claude plugin validate --strict at HEAD, and release.yml's line-shaped sed for "version": "X.Y.Z", (plus tests/test_action_version_pin.py::_PLUGIN_PIN_PATTERN) still matches after version moved from key #2 to key #4 — so the release bump is unaffected. The eval run path is untouched; nightly eval suites are unaffected. (trigger: .claude-plugin/marketplace.json)

Harness & Lint Improvements

Static checks (lint / type):

• [ce-lint] CE040 — cross-manifest metadata parity (new whole-tree rule). New module tests/lint/plugin_manifests.py, wired as tests/test_custom_lint.py::TestCE040PluginManifestParity (NOT a BaseRule in tests/lint/runner.py — that runner is AST-only over .py; precedent: CE026-CE031, CE033, CE035). It json.loads /Users/religa/src/coder_eval/.claude-plugin/marketplace.json, selects the plugins[] entry whose name == "coder-eval", json.loads /Users/religa/src/coder_eval/plugins/coder-eval/.claude-plugin/plugin.json, and asserts equality on the derived shared key set — set(entry) & set(manifest) minus two explicit allowlists (ENTRY_ONLY = {"source", "category", "tags"}, MANIFEST_ONLY = {"$schema", "version"}) — rather than a hardcoded 8-name list, so a newly added shared key (the failure mode this PR just demonstrated by growing the surface 3 -> 8) is guarded the day it lands, and a deliberately divergent key must be added to an allowlist with a reason comment. Slots beside tests/test_action_version_pin.py, which already reads plugin.json for version and exists verbatim because "nothing asserted it". Prevents: Finding 1 (medium, .claude-plugin/marketplace.json): 8 byte-identical hand-synced fields with no parity guard. It would have failed on origin/main, where keywords was already drifted ("skills" vs "claude-code-skills") — the browse-time copy describing the product differently from the installed copy. claude plugin validate --strict in the plugin-validate job cannot catch this: it validates each file against its own schema and never compares the two.
• [ce-lint] CE041 — $schema pins must be vendored and actually validated (new clause in the same tests/lint/plugin_manifests.py). Vendor the two declared schemas under tests/lint/schemas/claude-code-marketplace.json and tests/lint/schemas/claude-code-plugin-manifest.json, keyed by a SCHEMA_URLS: dict[str, Path] map. The rule then: (a) fails if a manifest's $schema value is not a key of that map (catches a typo'd or silently swapped URL — nothing in tests/, .github/, or Makefile mentions schemastore today, so the pin is editor-only decoration); (b) runs jsonschema.Draft7Validator (already a runtime dependency — no new dep, no network) over each manifest against its vendored schema; (c) reports any top-level key the schema does not declare at that level unless it is in an UNDECLARED_KEYS_OK allowlist with a reason string. Clause (c) is what records, in code, that displayName — the key this PR adds at plugins[] and at plugin-manifest top level — is accepted by claude plugin validate --strict but appears in neither pinned schema outside channels[], and passes today only because additionalProperties is unset. Prevents: Finding 2 (low, .claude-plugin/marketplace.json:2): a declared schema contract that nothing checks, plus the demonstrated disagreement between the pinned schemas and the runtime validator over displayName. Fully offline and deterministic, so it belongs in make lint/make verify, not in a network-gated job.
• [ce-lint] CE042 — discovery strings must live on the fields that are actually searched (clause 3 of tests/lint/plugin_manifests.py). Encode the client's real browse/install predicate as data: SEARCHED_FIELDS = ("name", "displayName", "description") (case-insensitive substring match, per the filter observed in Claude Code 2.1.247). The rule then asserts (a) every term in a repo-level DISCOVERY_TERMS tuple (skill, trigger, eval, ci, benchmark) appears as a substring of the concatenated searched fields of BOTH manifests, so a future description rewrite cannot silently drop a term the team believes it is ranking on; (b) every keywords/tags value that is not such a substring must be listed in an INERT_TERMS allowlist whose comment states that the field is not consulted by the in-client search — the allowlist entry is the warning to the next contributor, delivered at the moment they edit the array; (c) len(description) <= MARKETPLACE_DESCRIPTION_MAX_CHARS (start at 220; current is 141, observed median across installed marketplaces is 135), mirroring the SKILL_LISTING_BUDGET_CHARS precedent at tests/test_custom_lint.py:1249 including its "raising the ceiling is allowed in a commit that says why" convention. Prevents: Finding 3 (low, .claude-plugin/marketplace.json:21): 12 new keywords/tags strings that no discovery surface reads, with nothing to stop the next contributor tuning them expecting search impact. Also future-proofs the two levers that DO work (displayName, description) against an unnoticed regression.
• [ce-lint] CE044 — the listing lede is SSOT; the prose ledes are derived (generated-surface pattern). Treat plugins/coder-eval/.claude-plugin/plugin.json's description the way mkdocs.yml's nav: is treated for the index tables (CE028) and the models are for reference/criteria.md (CE033): add a make plugin-lede target that writes the manifest description into marker-delimited blocks (<!-- plugin-lede:start --> / <!-- plugin-lede:end -->) in plugins/coder-eval/README.md (the lede at lines 3-4) and the description: frontmatter of docs/PLUGIN.md (lines 2-5), and a rule that re-renders and diffs — reusing the shared write/diff engine both generated-surface checkers already route through (tests/test_custom_lint.py:2765). If full generation is judged too heavy for two paragraphs, the cheaper mechanical form is: assert the manifest description's first clause (text up to the em dash, normalized to lowercase and stripped of punctuation) is a substring of the README's first two prose lines and of the docs/PLUGIN.md frontmatter description. Prevents: Finding 4 (low, plugins/coder-eval/.claude-plugin/plugin.json:6): commit 8d7eba9 inverted the pitch to lead with skill-activation testing in both manifests, but plugins/coder-eval/README.md:3-4 and docs/PLUGIN.md:2-5 still lead with the old ordering — so a user who browses the marketplace and clicks through gets the pitch inverted. plugins/coder-eval/README.md is already a governed SKILL_DOC_SURFACES entry (tests/test_custom_lint.py:1238), so the surface list needs no new concept.
• [ce-lint] CE042 clause (d) — the listing must name the out-of-band prerequisite whenever the skills defend against it. Mechanical and non-semantic: if any plugins/coder-eval/skills/*/SKILL.md contains the defensive sentence fragment did not install the CLI (it appears in init/SKILL.md:17, check-skill/SKILL.md:61, task/SKILL.md:20), then both manifests' description must match /(coder-eval )?CLI|separately installed/i. The plugin's own skills already treat this as the top user surprise and docs/PLUGIN.md gives it a bold ## Prerequisite heading; the rule just forces the same disclosure onto the one surface a prospective user reads before installing, and keeps it true if the prerequisite is ever removed (drop the sentence from the skills and the clause goes quiet on its own). Prevents: Finding 5 (low, plugins/coder-eval/.claude-plugin/plugin.json:6): an imperative capability lede for a plugin that ships no runtime, with ~80 chars of listing budget still unused. Flagged as the most opinionated of the proposals — if the team prefers not to legislate marketing copy, drop this clause and keep CE042 (a)-(c).

Harness improvements (not statically reachable):

• Nightly re-fetch + diff of the vendored $schema copies. Add a step to the existing nightly verify-published-action.yml (which already does published-artifact parity checks on a schedule) that curls https://json.schemastore.org/claude-code-marketplace.json and .../claude-code-plugin-manifest.json and diffs each against tests/lint/schemas/*.json, failing (or opening an issue) on drift or on a non-200. Keep it OUT of make verify so PR CI stays offline and deterministic. Why not static: Needs network and reaches a third-party artifact that changes without a commit here. CE041 can prove the manifests match the vendored schema; only a live fetch can prove the vendored schema still matches what the declared URL serves, or that the URL has not 404'd. Prevents: Finding 2 — the pinned schema URLs are never resolved by anything, so a later-dead or renamed upstream schema would ship unnoticed.
• Post-publish marketplace parity check on the served artifact. In the same nightly/post-Release workflow, fetch https://raw.githubusercontent.com/UiPath/coder_eval/main/.claude-plugin/marketplace.json (what /plugin marketplace add UiPath/coder_eval actually consumes) and re-run the CE040 shared-key comparison against the published plugins/coder-eval/.claude-plugin/plugin.json from the same ref, plus assert the entry's version-adjacent pins agree with the released PyPI version. Why not static: A repo-local lint proves the working tree is consistent; it cannot prove the copy users are served is. Release-path seds, a reverted commit, or a hand-edit landing on main outside a PR would all pass local lint and still leave the front door describing itself two ways. Prevents: Finding 1 — the drift is user-visible precisely on the published copy (browse-time description/keywords vs. installed plugin.json).
• make plugin-lede target + a CONTRIBUTING runbook line. Ship the generator half of CE044 as a first-class target next to make docs-indexes and make plugin-reference, list it in the CLAUDE.md "regenerate a generated surface" block, and add one line to CONTRIBUTING's release/plugin section: changing the listing pitch means running make plugin-lede and re-reading the two prose surfaces. Why not static: The lint rule can only fail the build; the dev-workflow gap is that today there is no one-command way to ripple a pitch change, so the reviewer's suggested fix is a hand-edit in three files — exactly the shape that produced the drift. Prevents: Findings 1 and 4 — both are hand-sync surfaces with no generator.
• Pin and record the Claude CLI version the plugin-validate job validates with. .github/workflows/pr-checks.yml:218-243 installs @anthropic-ai/claude-code unpinned, so the meaning of claude plugin validate --strict (and of the browse/search predicate the CE042 discovery clause encodes) silently follows upstream. Install a pinned version, echo claude --version into the job summary, and put a dated comment next to SEARCHED_FIELDS in tests/lint/plugin_manifests.py naming the client version the predicate was read from. Why not static: The searched-field set lives inside a third-party binary and can change between releases; no repo-local check can introspect it. The best available guard is making the assumption dated, attributed, and observable in CI output so the next person can re-verify instead of trusting a stale comment. Prevents: Finding 3 — the whole argument that keywords/tags are inert rests on one observed client build; without a recorded version the conclusion decays into folklore.

Top 5 Priority Actions

1. No finding in this review can change a task's score or final_status for identical agent output — the diff is plugin-manifest and docs metadata only, so treat every item below as maintainability work rather than correctness risk.
2. Add a ~15-line parity test beside tests/test_action_version_pin.py that loads both manifests and asserts the 8 shared keys (name, displayName, description, author, homepage, repository, license, keywords) are equal between .claude-plugin/marketplace.json:12-31 and plugins/coder-eval/.claude-plugin/plugin.json:3-19, since this change grows the hand-synced surface from 3 fields to 8 and the pair already drifted on keywords on main — scope it to field equality, because plugin-validate in .github/workflows/pr-checks.yml:218 already schema-validates each file on its own.
3. Ripple the new skill-activation-first lede from plugins/coder-eval/.claude-plugin/plugin.json:6 into the two prose surfaces still carrying the old ordering — plugins/coder-eval/README.md:3-4 (a governed SKILL_DOC_SURFACES entry) and the docs/PLUGIN.md:2-5 frontmatter — so a user clicking through from the marketplace listing does not meet the inverted pitch.
4. Append a short clause naming the separately installed CLI to the shared listing description at plugins/coder-eval/.claude-plugin/plugin.json:6 and .claude-plugin/marketplace.json:15 (139 chars used, well under the 586-char observed ceiling), because four of six skills open by defending against exactly this surprise (plugins/coder-eval/skills/init/SKILL.md:17).
5. Either add a network-optional test that validates each manifest against a vendored copy of its declared schema, or drop the unverified $schema pins at .claude-plugin/marketplace.json:2 and plugins/coder-eval/.claude-plugin/plugin.json:2 — neither schema declares the displayName key this change adds at the level it is used, so the pin passes only because additionalProperties is unset.
6. Record in a commit note or contributor doc that keywords and tags (.claude-plugin/marketplace.json:21-30) are inert for in-client discovery — the browse filter matches only name, displayName, description and the marketplace name — so the next contributor tunes description instead of a field that cannot affect search.

Stats: 0 🔴 · 0 🟠 · 1 🟡 · 4 🔵 across 7 axes reviewed.